Networked video devices

ABSTRACT

A collaborative information system.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. patent applicationSer. No. 10/906,754, filed Mar. 4, 2005, which claims the benefit ofU.S. Provisional App. No. 60/550,301, filed Mar. 4, 2004 and U.S.Provisional App. No. 60/629,726, filed Nov. 19, 2004.

BACKGROUND OF THE INVENTION

The present invention relates to collaborative information systems.

Users watching video content on a television are not always availablewhen a particular show is broadcast. In order to permit watching a videoat a later time or otherwise the video, personal video recorders havebeen developed which permit the user to record the video content on astorage device and play it at a later more convenient time. Existingpersonal video recorders include the capability of creating and updatinga user profile of the user's viewing habits. Based upon this userprofile, the system may filter potential content and provide the userwith recommendations. Accordingly, for a particular television set andan associated personal video recorder in a user's home, the personalvideo recorder may create a user profile which is used to recommendpotential video content to the user.

Some existing Internet based shopping based websites, particularly thoserelated to selling books, track in some manner the previous selectionsof different users. The website may track a trend such as the userstendency to purchase a particular first book together with a particularsecond book. The data collected may then be used to providerecommendations when another user indicates a preference to purchaseeither the first or second book by presenting the other book as being ofpossible interest. While the information may be of possible use, thereis no indication of the source of the data or otherwise how such arecommendation was made, or the identification and background of thoseother users. In many cases, the recommended book may be of little to nointerest to the particular user.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a context diagram.

FIG. 2 is a presence technology system.

FIG. 3 illustrates processing nodes.

FIG. 4 illustrates viewer set collection types.

FIG. 5 illustrates viewer set characteristics.

FIG. 6 illustrates nested viewer set members.

FIG. 7 illustrates nested viewer set single entity.

FIG. 8 illustrates peer to peer technology.

FIG. 9 illustrates client server topology.

FIG. 10 illustrates peer to peer processor architecture.

FIG. 11 illustrates client server processor architecture.

FIG. 12 illustrates processing sequence.

FIG. 13 illustrates summarization scenarios.

FIG. 14 illustrates content context collaboration input.

FIG. 15 illustrates content context collaborative input.

FIG. 16 illustrates an out/in preference filter description.

FIG. 17 illustrates a system with conversation communication channels.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT General Overview

Referring to FIG. 1, a video based system includes a user 100 whowatches or otherwise consumes audiovisual content provided to the userin some manner. In many cases the audiovisual content is provided on adisplay 102 which may be interconnected to a personal video recorder104. The personal video recorder 104 may be any type of device thatreceives a video signal or otherwise video content and stores the videocontent for later use, such as on a hard drive or optical storagemedium. In some cases, the personal video recorder may provide links orotherwise access to network locations from which the video is available,normally video on-demand. The display 102 can also have a networkconnection for communication with a network. Also, the display 102 canbe provided with or without an associated personal video recorder 104.In some cases, the video content in that case would be on other devicesor a server device, such as video on demand (VOD), TV broadcast, orInternet. For example, the TV may be an Internet Protocol (IP) capabletelevision for receiving data and/or video, together with broadcasttelevision, if desired. The video content 106 may be provided from anysource, such as for example, the content may be provided by physicalmedia 110, live video feed 112, computer generated video 114,interactive television 116; the provider of the video may be broadcastnetwork 118, video on demand 120, other backend source 122, Internetserver 124; and the distributor of the video may be cable 126, satellite128, terrestrial broadcast 130, and Internet 132. Information related tothe audiovisual content may be provided as program metadata 140 from anysource, such as for example, program website 142, primary reference(e.g., MDB) 144, metadata consolidator (e.g., TV Tome) 146, schedulingmetadata 148, creation metadata 150, and descriptive metadata 152. Otheruser's 160, 162, 164 (generally referred to as peers) may have access toat least some of the same (or similar) audiovisual content and some ofthe same (or similar) program metadata 140. The users 100, 160, 162, and164 may also have access to the same (or similar) enhanced services 166,such as for example, recommendations 168, and interactive TV 170.

Existing personal video systems are based upon the premise that aparticular personal video recorder is coupled with a single televisionfor recording and recommending content for a particular user(s) of thatdevice. Further, current personal video recorders are based upon apremise that upon the viewing habits of the user are sufficientlyeffective for making additional content recommendations. Therecommendations may be provided by the personal video recorder orotherwise from a service provider. In contrast to this model, thepreferred embodiment includes networking together multiple personalvideo recorders or otherwise multiple display devices (such as TV,computer, mobile phones, or otherwise) in order to share information inorder to achieve functionality that would normally not be availablewithout such interconnectivity. A network 180 interconnects multiplepersonal video recorders together, which may be in the form of theInternet, local area network, wide area network, wireless network,direct connection, telephone interconnectivity, wireless network, orotherwise. In addition, the system permits one to explore beyond theuser's own viewing habits so that the viewer can discover new content.Moreover, the system permits the user to enhance their ownrecommendations based upon other networked users.

In many cases, a group of users (e.g., a group of friends) areseparately and simultaneously viewing the same, different, or similarvideo content. In most cases, this separate viewing occurs without theother users being aware of each other, aware if others user arecurrently viewing content (“off line” or ‘on-line’), or otherwise awareof what content each other is viewing. In addition, in most cases thedifferent viewers are unaware of the content preferences of the otherviewers. In the event that a user was aware of the presence of the otherusers, such as whether they are currently viewing content or notcurrently viewing content, then the user may be able to use thispresence based information in the recommendation of suitable content.Preferably, the currently viewed content by a user is shared with otheruser's in a manner so that a user may obtain recommendations forsuitable content. In addition, other data from viewers currently viewingcontent may likewise be shared with other user's in a manner so that auser may obtain recommendations for suitable content.

To most effectively implement such a system, the users publish data toother users and likewise subscribed to data from other users within thepresence based collaborative recommendation system. A comprehensivecollection of publisher recommendation data is disseminated and anexpended/aggregated set of this recommendation data is received by thesubscriber. The recommendations made by the publisher may beautomatically generated by the system or otherwise provided manually.The recommendations may be filtered in any suitable manner for theparticular subscriber and for privacy concerns of both publishers and/orsubscribers.

The presence based system may arrange the publishers as sets of viewersin a variety of different arrangements. The recommendations originatingfrom the publishing viewers may include, for example, current programselection, noted selections, accumulated/refined program selections(such as based upon usage history), content selection, preferencedescriptions, and channel selections, where program and contentselection include program and content watching or otherwise recording.The recommendations may be filtered for minimum selection preferencecharacteristics, maximum selection preference characteristics, andsufficient continuity durations to prevent undesired channel-surfingrecommendations. In addition, the recommendations may be localized forthe subscriber's content availability and novelty. The recommendationsfrom multiple publishers may be combined in a collaborative manner toprovide, for example, statistical averaging, cross-group collaboration,interest indices, statistical metrics, context index weighting andaccumulated collaboration.

The presence based system may include content consumptionrecommendations. These recommendations may be provided from a user'spublishing viewer set and may refer to and describe recommended segmentsof a video content. In many cases, these segment or summary descriptionsare generated for recorded, and time—shifted programs, which may beannotated and edited.

The presence based system may include the viewer sets being organized ina variety of different ways. The viewer sets may each include differentprivacy and permission attributes. In addition, the viewer sets may beorganized as one-to-one, multiple viewers, or hierarchical combinations.Subscribers may join one or more viewer sets, each of which may have adifferent organization, privacy attribute, and permission attributes.The presence based collaborative system provides a social context tocontent selection and consumption, and enables sharing of contentselection choices and recommendations among users that belong to certainviewer sets.

The presence based system may provide a remote control device with thecapability of content selection, capture, and browsing. The remotecontrol device may be linked to a controller, such as a publishingviewer, or a source of program summarizations. The controller may shiftto different entities during a typical session. The subscribers that areconcurrently linked to the same controller in this fashion concurrentlyconsume and co-view the same program rendition. The publisher programtimepoint/offset may be shifted to accommodate concurrent browsingcontrol.

The presence based system may include conversational communication, andinclude content-integrated virtual viewer environments. Conversationalcommunication may include instant messaging, voice over IP, video overIP, or videoconferencing over IP. To assist with navigation, theenvironment may include onscreen pointers, onscreen markup pens, andother presentation and navigation tools. In addition, the conversationalcommunication may include avatars and avatar-like self representations.The conversational communication may be triggered by any event, such asthe system, plug-ins, on occurrence of an event, or otherwise manually.

The presence based system may include supplementary relatedrecommendations. Supplementary recommendations include programselections, media delivery options, program summarization descriptions,and related descriptive material. Supplementary recommendations may beprovided by cross-group viewer sets or commercial or other affiliatedorganizations.

System Configuration

Referring to FIG. 2, the user 100 (also referred to as target viewer)may join 210 other users 220 (also referred to as peer viewers) acrossthe network 180. In this manner, the target viewer 100 may exchange datawith the peer viewer(s) 220. The target viewer 100 may exchange datarelated to videos with the peer viewers 220 in a variety of differentmanners, described later. Based upon joining with other viewer(s) thetarget viewer 100 establishes an inbound path 230 for recommendations orotherwise data to be received from other viewers 220 (or a server).Based upon joining with other viewers 220 the target viewer 100establishes an outbound path 240. Based upon requests or otherwise, thetarget viewer 100 may provide recommendations or otherwise data to otherpeer viewers 240 (or a server). It is noted that the target viewer 100may join 210 multiple different peer viewer sets which may be used forthe same or different recommendations 230 and 240.

In order to achieve an efficient implementation, it is desirable toinclude a framework within which data can be exchanged in an efficientmanner. In addition, the framework should include an effective manner inwhich to find, retrieve, and subscribe to changes in presenceinformation (e.g., “online” or “offline”) of other users.

Referring to FIG. 3, an exemplary system by which recommendations may beprovided and received is illustrated. It is noted that the “upper” and“lower” paths may be essentially mirror images of one another so thatwhen the target viewer publishes to peer viewers and when peer viewerspublish to the target viewer the same processing sequence may beperformed. While FIG. 3 illustrates each peer as a separate publisherand subscriber, in most cases the publisher and subscriber are typicallycomprised of both. In addition, the recommendations may be determined bythe target viewer 100, based upon received data, or otherwise therecommendations are provided to the target viewer.

The viewers (target or peer) are at the endpoints of the system. In somecontexts, the human user and the local content manager device (e.g.,personal video recorder) are separate entities. For instance, thepersonal video recorder device may be “online” while the human user isnot present (“offline”), the personal video recorder device may be“online” while the user is “online,” the personal video recorder devicemay be “offline” while the user is “offline,” or the personal videorecorder device may be “offline” while the human user is “online” withthe content being provided from a different device or server. In manycases, the presence of either the user, the content management device,or both may be important.

As the user views programs or otherwise navigates through content, oneor more parts of the system may create or otherwise provide, informationregarding interrelationships between different programs and thesuitability of particular content for a particular viewer. In addition,interrelationship information may be explicitly provided by the user, ifdesired. In some cases, the information provided or created may be inthe form of TV-Anytime compliant metadata. The outbound generator 310receives information (which may be “raw” or “processed”) and in responseprovides information, which may be in a generally unprocessed format, ofpotential recommendations for video content.

In many cases, the data provided by the content management device may beunsuitable in its current form for effective use by others. For example,the data may include personal information, proprietary data which wouldbe of little to no use for others, or general data that is not usefulfor sharing with others. In some cases, the user may select one ofmultiple different privacy thresholds, and as such different data ordifferent amounts of data may be shared with others.

An outbound processor 312 may perform filtering of the data from theoutbound generator 310 based upon the privacy settings, personalinformation, proprietary data, etc. to remove unnecessary data frombeing forwarded to others. In addition, the outbound processor 312 mayaugment the data from the outbound generator 310 with additionalinformation to make the data more suitable for others to use. Also, theoutbound processor 312 may format the data to a format suitable forother devices. Also, in some cases there may be a large amount ofdifferent data sets being provided to others, in this case, it may bedesirable to reduce the amount of data to a more manageable amount. Thisis of particular value when the receiving device has limitedcomputational resources. In such case, the outbound processor 312 mayaggregate different data together to reduce the amount of data. Further,the outbound processor 312 may perform the initial routing of the datato one or more other devices or users. In this manner, appropriate datawill be provided to appropriate device(s).

Within the system there are multiple data sets that may berepresentative of similar data, such as data from two separatepublishers. In this case, it may be useful to further reduce the amountof data being transmitted to target viewers by removing redundancy orotherwise reducing the amount of data. A collaborative pre-processor 314may perform filtering operations on the data from a plurality ofdifferent publishers, primary to reduce redundancy.

A collaborative processor 316 may perform aggregation and collaborativefiltering functionality, if desired, on the data from a plurality ofdifferent publishers.

A collaborative post-processor 318 may perform filtering and routingfunctions, if desired, on the data from a plurality of differentpublishers.

An inbound processor 320 may perform filtering and augmentationfunctions.

A presentation and interaction processor 322 may format therecommendations for presentation to the user and provide a structure forthe user to interact and act upon the recommendation.

It is to be understood that the components of the system may bedistributed across the network in any suitable manner. In general, it ispreferable that the outbound processes, the inbound processor, and thepresentation and interaction processes are co-located with the viewer.Generally, both the outbound and inbound processes are co-located withthe viewer. The collaborative processes (pre-, post- and base) may belocated at the viewers, or preferably centrally located as a server, orcombined in a peer-to-peer architecture.

Recommender

Referring to FIG. 4, a viewer set is the collection of peer viewers thatprovides collaborative recommendations to the target viewer. This may bereferred to as the publishing viewer set. An extension is that thetarget viewer may “join” a viewer set, and the combined viewer set(target viewer plus peer viewers) provides collaborative recommendationinput to the combined viewer set. This combined viewer set is a set inwhich all (or a plurality of) viewers are mutual publishers andsubscribers.

To receive recommendations the target viewer joins one or more viewersets. The viewer set may be an ad hoc collection that is unique for eachtarget viewer, or it may be organized into a collection that is commonfor all viewers in the set.

Referring to FIG. 4, the three principal viewer set collection types areillustrated. These collection types can be approximately categorized bythe anticipated size of the viewer set. However, the actualimplementation may result in a continuum of the attributes described foreach of the types.

Buddy Roster

A buddy roster viewer set 400 is normally created in an ad hoc fashion.This is where the target viewer joins directly to one or more “friends”or known parties. Generally, this is a small collection of close friendsor acquaintances. For a target viewer who is new to the system, thebuddy roster viewer set 400 may be the most comfortable type of set touse. Generally, the buddy roster viewer set 400 is private in membershipand private in visibility to others outside the target viewer.

Group

A group viewer set 410 may be organized as a common group where manyusers may join the same group. Commonly, the group viewer set 410 is amedium sized collection of acquaintances that are close friends or havesome common interest. Generally, the group viewer set 410 is private inmembership and private in visibility to others outside the group viewerset 410.

Community

A community viewer set 420 is organized as a common group where any usermay freely join the group. Typically, the community is a large sizedcollection of distant acquaintances or unfamiliar users that normallyhave a common interest. Generally, the group is public in membership andpublic in visibility to everyone.

Mixed Collection

Typically, there will be an evolution in the viewer's adoption of thedifferent viewer sets provided by the system. The viewer will start withone or two buddies in a buddy roster 400, and then expand to join agroup 410 or a community 420. Eventually, the viewer may have multipleviewer sets, with different policies (described below), which may haveoverlapping intersections, and the viewer will be comfortable with themall being active simultaneously.

Policy Details

Viewer sets may have many configuration settings and policies forjoining and viewing different sets including:

-   -   Joining the viewer set (membership) may be unrestricted or        restricted as defined by the viewer set administrator. (E.g. via        pre-defined list, or via a request/approval process.) Viewer        sets may be open or closed for new members.    -   Viewer set details may be visible or hidden to non-members. For        instance, the identity of members and the policies of the viewer        set may be visible or hidden. The viewer set may list its set        name for other viewers to request admittance to. Alternatively,        the viewer set name and all details about the set may be hidden        to increase privacy.    -   Recommendations generated by members of a viewer set may have        some details visible, hidden, or removed. For instance, the        sender identity and details about his viewing environment may be        visible, hidden, or removed. Anonymous recommendations may have        a variable degree of privacy, to allow some personal context        information to be visible even if the precise identity is        hidden. For instance, the viewer's available content provider        services (e.g. cable provider package) may be pertinent to the        recommendation processing, and thus be viewable.    -   Viewer sets may include default minimum and default publication        options. For example, a public community such as the Bay Area        Retired Persons Viewer Community may require that all members        provide anonymous unedited viewing recommendations to the        community at large. This is helpful to inhibit viewers from        editing or filtering their viewing data which would skew the        collaborative results. These default and minimum publication        options can be applied to individual viewers or to viewer sets        that are nested into other sets. The policies are generally        attributed to organized viewer sets. However, they may be        attributed to individuals who utilize a buddy roster type viewer        set, instead of an organized viewer set.

Recommendation Characteristics

The three principal collection types may be ordered by size of viewerset. Prototypically, these collection types have set characteristicsthat show trends as depicted in FIG. 5. These characteristics aregenerally increasing or decreasing as shown. However, this is not afixed relationship.

-   -   Acquaintance 510 is the degree of familiarity between viewers        within the viewer set. Buddy rosters are typically close        friends, while large communities tend to be made up of remote        acquaintances or even strangers.    -   Discovery 512 is the ability for a viewer to receive        recommendations that are outside his typical selection        preferences. A recommender system that is based solely on a        viewer's content selection preferences or on the preferences of        his close (similar) friends has less opportunities for        discovery. Conversely, a recommender system that includes many        viewers of low acquaintance has greater opportunities for        discovery.    -   Coordination 514 is the ability for viewers to interact with        each other. It is also the ability for viewers to manipulate        other viewers' environments.    -   Privacy 516 is the measure of information and identity hiding.        Typically, in a large community, the users would want their        identity to be anonymous, and to limit the availability of        personal information.        Independent of the viewer set type, viewers of the system may        mandate heightened privacy levels until they are more        comfortable with the system. This heightened privacy may be the        default during initial configuration.

Hierarchy

Viewer sets may be nested, such that the members of one viewer set arejoined into another set, thereby forming a hierarchy of viewer sets.This may provide the following benefits.

-   -   The collaborative processors and the user interfaces can blend        the recommendations from the parent superset with the child        subset. In this way, the nested subset can receive the diversity        of recommendations from the parent superset while permitting the        relaxed privacy restrictions within the subset. This provides        enhanced discovery.    -   Other administrative and regional benefits result from the        hierarchical viewer sets.

Examples of scenarios in which viewer sets may be nested are describedbelow.

-   -   Nested friends (groups)—A large group of friends may form a        broad viewer group and smaller subgroup of closer friends may        form within this larger group. For instance, a group of a dozen        friends with common interests may form a viewer group, and then        two or three best-friends may form a subgroup within.    -   Special interest (communities)—A large community of viewers with        a common interest may form a semi-open viewer community, and        smaller sub-communities may form within this larger community.        For instance, the Bay Area Aviators Viewer Group (or Retired        Persons, or Historical Society) may be formed. This community        may permit and promote sub-communities or chapters to be formed.        These sub-communities may even support additional nesting of        sub-communities.    -   Academic (groups/communities)—A teacher or seminar speaker may        form a semi-open group for a particular subject. However, this        type of group is probably not useful except where multiple        simultaneous viewing choices are applicable to the subject.

Nesting Architecture

The viewer sets may be nested in many ways. For example, each of themembers of the nested viewer set may be individually projected into theparent superset, maintaining their individuality. Alternatively, thenested viewer set may be projected en masse into the parent superset,and the nested viewer set becomes a single composite entity in theparent superset.

-   -   Nested Members—Referring to FIG. 6, the individual members may        be entered into the parent superset. Viewers B and C are members        of the Close Friends Group which is nested into the Acquaintance        Group. Viewers A, B and C are projected through the appropriate        privacy filters as members of the Acquaintance Group, but        viewers B and C are able to share recommendation and interact        with the relaxed privacy constraints of the Close Friends Group.    -   Nested Sets—Referring to FIG. 7, the nested viewer set may be        entered as a single entity into the parent superset. This form        of nesting may provide organizational benefits as well as the        privacy benefits of the nested members form.

Viewer Profiles

Users of the system may have personal profiles. The system may maintainidentification and other presence information. When viewers joinorganized viewer sets (groups or communities), additional member profileinformation may be maintained for the user. Viewers that are members ofmultiple viewer sets may have a profile for each set. All profileinformation may be classified as visible or hidden to all, to the othermembers of the viewer set, to the administrator, and even to the targetviewer. Some examples of viewer profile information is described below.

-   -   Viewer identity, including name, handle, password, address,        e-mail, IP address, etc.    -   Home audio and/or video configuration.    -   Content provider details.    -   Inferred statistical data about the viewer and his        recommendations. This may include viewing durations of        particular video content, recommendation accept/reject rate,        usage history, auto-profiled preferences, search history, etc.    -   Configuration settings for processing preferences may be part of        the profile, and may be managed at the viewer location and/or        they may be managed as part of the viewer set. This may include        subscription and publication settings, content preferences,        permission/access control lists, etc.

Architecture

The processing nodes for the system, depicted in FIG. 3, can bedistributed and combined in several configurations. Two commonconfigurations are peer to peer and client server.

-   -   In the peer to peer configuration, each viewer is connected        directly to the other members in his viewer set, as depicted in        FIG. 8. Each connection has an inbound and outbound path,        corresponding with the subscribed and published recommendation        paths. These two paths are somewhat independent. For instance,        viewer C has subscribed only to viewer D for recommendations,        but he is publishing to both viewers B and D.    -   In the client server configuration, each viewer is connected to        a central or distributed server. This basic configuration is        depicted in FIG. 9.

Processor Distribution

The topology of the system determines where the processing nodes shouldbe located. The two common configurations are described below.

-   -   The peer to peer topology may result in a processor architecture        as depicted in FIG. 10. The collaborative processing is        duplicated for each viewer. Inbound processing is expanded to        perform the collaborative processing. Outbound processing is        relatively light, allowing for a thinner publishing component.        (An alternative approach is to expand the outbound processing        operations to include the collaborative processing, which would        result in a thinner subscribing component.) However, shifting        the processing to the inbound or the outbound side doesn't        reduce the total viewer processing because both paths are        included in a normal viewer.    -   The client server topology generally results in a processor        architecture as depicted in FIG. 11. The collaborative        processing is centralized in the server. Inbound and outbound        processing (subscribing and publishing) are thinner because of        the centralized collaboration.

Mixed Architecture

Each viewer set should be supported by an architecture, such as peer topeer or client server. However, a viewer may belong to multiple viewersets which may have different architectures. Even if the multiple viewersets have the same architecture, they may have different serviceproviders. The result is that the viewer is connected to multipleinfrastructures. The seamless combination of multiple infrastructuresmay require that the viewer component (combined publisher andsubscriber) be able to connect to the various infrastructures, andmanage the various viewer sets, preferably in a common integratedformat.

Processing Sequence

Processing operations may be distributed amongst the processes in avariety of combinations. The table below summarizes the typicalassignment.

Outbound Outbound Pre-Collab Collab Post-Collab Inbound PresentationGenerator Processor Processor Processor Processor Processor &Interaction Manual/Automatic X Generator Manual/Automatic X X PrivacyFilter, Edit, Route X Target Filter Continuity X X X X CollaborativeFilter X Target Filter X X X Preferences Target Filter Weighting X X XLocalization X X X Presentation X & Interaction

Recommendation Types

The Outbound Generation Process 310 provides the “raw” recommendationinformation. This may be automatically inferred from the viewer'scontent navigation and program selection, or it may be manuallygenerated by explicit user input.

Referring to FIG. 12, the program selection and channel selectionrecommendation types are forms of more general content selection.Program summary recommendation type is a form that supports contentconsumption.

Program Selection Recommendations

The publishing viewer generates recommendation information that is usedto help the subscribing viewer to make specific program selections. Thisinformation may be plain or aggregated program listings, or it may befilter descriptions that ultimately yield program listings.

Currently Viewed Program Selection

The program that the publishing viewer is currently watching is a strongselection recommendation. The outbound generation process 310/312automatically posts the current program recommendation (through thecollaborative processor) to all suitable subscribers. An updaterecommendation is posted when the program changes. Program changes mayoccur due to:

-   -   Viewer actions such as turning on/off the television or changing        channels;    -   Normal broadcaster change of program;    -   Local media players reach the end of the program;    -   Schedulable playback device starts playing a new program.        Commercials and intermissions are preferably not considered        program changes. Additional context information (e.g. available        content), and filtering (e.g. surfing and privacy) may be        provided. The core raw information is the current program being        viewed. The currently viewed program selection may be a direct        result of a selection chosen from the publishing viewer's        content preference filter. In this manner, the publishing        viewer's content preference filter is indirectly propagated to        the subscribing viewer set.

Unviewed Program Selections

The specific program that the publishing viewer is currently watchingmay be the strongest program selection recommendation. However, theremay be additional programs that the viewer found to be close runners-upwhen he was searching for his program. These secondary selections may bevaluable recommendations to some subscribing viewers.

-   -   Using the Manual Generator 1210 the publishing viewer may        examine his available content list (possibly through a search        list) and he may consider some programs to be good selections,        even though he may not choose to watch them presently. In these        cases, he can actively denote these programs (within his program        selection user interface 1220) for recommendation publication.    -   During the search process, some programs may turn up which are        very similar in metadata description to the final selected        program. This determination may be done automatically by the        automatic generator 1230, and it may also pass through dedicated        selection preference filters 1240. For example the viewer may        have generated a search list of several vampire movies. He        selects and starts watching one of them without previewing the        others. The currently viewed program will be posted as a primary        recommendation. The other vampire movie possibilities should        optionally be qualified by the secondary recommendation        selection preference filters, and then be posted as secondary        recommendations.    -   The publishing viewer may have access to a different recommender        source, for instance a suggestion list. The close runner-up        programs which are not selected may be posted as secondary        recommendations.        The appropriate filter and preference weightings are applied to        the secondary recommendation types, to ensure only higher        correlated selections are presented.

Usage History

The publishing viewer is consuming content through a device which hascontent management functionality. Logging of UsageHistory 1250 is acomponent of content management. There are two primary ways in which theUsageHistory 1250 can be used to generate program selectionrecommendations.

-   -   The outbound generation processor may aggregate the UsageHistory        log, and use the detailed user action events to form a compiled        program selection recommendation that has a refined preference        weighting description.        The user action data is information that does not typically get        published to the subscribing viewers. For example, events like        pause/resume and rewind/play indicate that the viewer was        especially interested in the program. The user's consumption        actions may automatically create and refine content preference        filters. Similarly, the user's consumption actions, as logged in        the UsageHistory table, will create and refine specific program        selection recommendations.    -   The UsageHistory can be used to generate recommendations as a        form of catch-up. In the initialization of the system for a        publishing viewer, the UsageHistory can be used to upload any        recent usage that was missed while unconnected. Alternatively,        in the initialization of the system for a subscribing viewer,        the UsageHistory can be requested from all publishing viewers,        and the downloaded usage information will allow the system to        catch-up. This may not necessarily be a form of instantaneous        recommendation, and it could be implemented in a system without        presence technology functionality.

AD HOC Section Preference

The list of program titles (or other defining descriptors) is a resultof the publishing viewer's selection process. Taking it back a step, asubscribing viewer set can share the actual process of selection.

Ad hoc selection preference filtering uses temporary content filters tosearch for desired content. For instance, if a buddy uses severalkeywords or genre descriptions to search for programs, these temporaryfiltering descriptions may be meaningful only at that point in time. Thenext day, or even later that same evening, these ad hoc descriptions maybe abandoned. If they are not abandoned, they may become part of thefixed selection preference filtering descriptions.

Fixed or semi-permanent selection preference filtering descriptions canbe shared through offline (non-instantaneous) channels, or they can bemanaged with the system. However, ad hoc selection preference filteringdescriptions should be “instantaneously” shared. The publishing ofselection preference filtering descriptions, whether they are ad hoc orfixed, may be done using a standardized description structure, such asMPEG-7's Filtering And Search Preferences description scheme, orTV-Anytime.

Channel Selection

A viewer group may generate a content preference listing for broadcastchannels or other content delivery channels. The channel selection maybe a coarse or fine recommendation type. A coarse channel selection isaccumulated over multiple viewing cycles, while a fine channel selectionis an instantaneous notification. (In the extreme case of instantaneousnotification, the channel recommendation is essentially another type ofdescriptor for the currently viewed program selection.) Channelselection recommendations may be dependent on the time of day or week.

Program Summary

The publishing viewer may generate a summarization description for aselected program. The description may include skipping ahead, repeatingclips, playing at slow or fast rate, and other browsing operations. Theprogram summarization description is more meaningful for content thatcan be non-linearly navigated. This may be content that was recorded,time shifted (e.g. TiVO live delayed TV), or that is available in localmedia. The content may have consistent timestamp or offset informationavailable to permit accurate browsing (transport) control.

The non-linearly navigable content may be available to both thepublishing and the subscribing viewers, although it doesn't have to bethe same run of the content. For example, the publisher may generate thesummary on an early run of the program, and the subscriber may consumeit on a later run that he records or time shifts. Summarizationdescriptions may be automatically generated by monitoring the publishingviewer's actions or by extraction from his UsageHistory log.Alternatively, the descriptions may be generated by a personalized(custom) or third party summarization application.

Summarization descriptions would be especially useful to, summarizesports programs, or summarize heterogeneous programs (e.g. the news isheterogeneously composed of weather, sports, etc.), or to skipcommercials. FIG. 13 depicts sports and news scenarios wheresummarization may be valuable.

SUMMARY

If the publishing viewer is watching and summarizing a time shiftedprogram, then the subscribing viewer may be able to use the summaryimmediately if he is also time shifting the program. This is a presencetechnology based way to watch the program. The viewer has“instantaneous” summary updates which enable him to watch the samesummary of the program at the same time as the recommender.

If the subscribing viewer activates the summary for playback on arecorded or local media, then the instantaneous nature of the summaryupdates may not be utilized in some cases. However, the presencetechnology is still providing value because it supports a sense oftimeliness to the published recommendations and summarizationdescriptions. As these inputs are received by the viewer's system, hegets an instantaneous notification. This may trigger the viewer to goand view the program and summary, or it may trigger him to reserve itfor a future viewing.

While the publishing viewer is watching and navigating the program, hemay edit or manually generate summary items. He may also annotate thesummary. This may be valuable to special interest groups such asteacher/student or political interest groups.

Published Data Elements

The recommendation types listed above (program selection, channelselection, and program summary) should support the following elementsand concepts.

-   -   Recommendation item details—uniquely identifying description        that is appropriate for the type of recommendation.        -   Program selection details—details that will enable location            of the given program. This may include program title,            subtitle, episode, content channel, creation and delivery            timepoint, etc.        -   Content channel details.        -   Summarization description—summary details (see MPEG-7 schema            or TV-Anytime)    -   Degree of preference—e.g. preference value in MPEG-7 or        TV-Anytime. By default, a certain positive degree of preference        is assigned or assumed about recommendations, but there will be        occasions when a recommendation deserves special preference        weighting. For instance:        -   An especially high preference program may be designated with            a higher than normal preference.        -   An urgent program (e.g. live broadcast) may be designated            with a high preference weighting, or a time condition may be            inserted (e.g. in the preference condition description of an            MPEG-7 message), or some other appropriate indicator may            designate urgency.        -   A negative preference or aversion may be designated by            negative preference weighting or similar.    -   Recommendation source type—description of the publisher's        generation source. This may include attributes of        primary/secondary, manual/automatic, surfing/EPG/recommender        selection, content-preferences/UsageHistory, etc.    -   Endorser position—description of how the item is currently being        consumed by the publisher. This may include the specific        timepoints/offsets into the program where the publisher is        currently watching the program, etc.    -   Endorser context—description of the environment from which the        publisher is making the recommendation. This may include fixed        elements such as the AV setup and the content channels that are        available to the viewer. It may also include dynamic elements        such as other viewers present and any factors affecting the        publishing viewer's attention to the content selection and        consumption process.    -   Annotation—Comments, summary, emphasis, etc. for the        recommendation. May be present in various formats e.g. text,        audio, graphics, video.    -   Condition qualifiers—description of time, location, or device        conditions for the recommendation (e.g. the preference condition        of an MPEG-7 description). For instance, a publisher may        designate that a program selection is intended for the family        viewing time period. Or, a publisher may designate that a        program summarization description is intended for a portable        device

Outbound Processing

The publishing component may further process the outbound recommendationdata. This outbound processing includes editing, manually generating,filtering and localizing the recommendation. Some of these may beexclusively tied to the publishing component. The editing operations andthe manual generating operations will augment the recommendationinformation with user input, and so they should remain in the publishingcomponent. The remainder of the outbound processing operations may beperformed outside the publishing component. In particular, the filteringand localization operations may be performed in the next processingnode, the collaborative pre-processor stage. These relocatableoperations are described in this section. The collaborativepre-processor stage is described implicitly by these operations, andthat stage is not described further in this document.

Comfort and Trust

The user's comfort level and trust level with the system shoulddetermine the degree of automation and the degree of advanced featureavailability. Conceptually, a global comfort or expertise level settingmay be entered for the user. Based on this setting, some operationsshould be performed automatically and others may require manualinteraction. Similarly, this setting may determine the level of detailor sub operations that are revealed and available to the user. Ingeneral, privacy and comfort levels are different forms of trust.Privacy filters are covered later.

However, there is one attribute of privacy that is particularly relevantto the comfort level of the system. The degree of openness andvisibility that is offered to the user should be related to the level ofcomfort. The default user (beginner with low comfort) should only beoffered features that feel safe, and advanced users (more comfortable)may be offered more open features. For example, advanced users will feelmore comfortable with distributing more personal information, knowingthat the server will properly secure this information.

The publishing viewer may choose to mask or filter their outboundrecommendations with the manual privacy filter 1260. The user shouldhave the option to manually override and disable recommendationpublication with an interface 1270. This may be done for a singleprogram, for a definable period of time, or until further notice.

The user may automatically filter recommendations that meet apre-defined content criterion, or that occur in a predefined environment(e.g. time, location, or device). This automatic privacy filter may bebased on a dedicated privacy content preference filter description. Allof the publication privacy filters may be customized for each recipient.For instance, the user may automatically or manually filterrecommendations for a particular subscribing viewer or subscribingviewer set. This is especially important when the user has multipleviewer sets with varying privacy characteristics.

Privacy Filter

The publishing viewer may be notified of outbound recommendations, whichprovides him with real feedback about the automatic system outboundprocessing. After being notified, the publisher may have the opportunityto filter, edit, annotate or route the recommendations at function 1280with interface 1290. The publisher may filter the recommendation to passor stop it from being posted. Also, he may optionally stop or post allrelated postings for the current program. He may edit the preferenceweighting or other fields of the recommendation. The publisher mayannotate the recommendation with text or voice comments. The publishermay be able to route the recommendation by designating the targetsubscribers who should receive it. All of the outbound processing forannotation and amendment may be customized for each subscribing vieweror subscribing viewer set. For instance, the user may choose to disablemanual notification and amendment for a subscribed anonymous community,but enable it for a small familiar group.

Annotation

The publishing viewer may manually generate recommendations. During theselection process, the publisher may encounter programs that he doesn'tselect for himself, but that he wishes to publish as a recommendation tosome subscribing viewers. For instance, “I think that Terry would liketo see this program, even though I am not watching it.” While consuminga selection, the publisher may wish to publish the current programselection recommendation to a viewer that might otherwise not receivethe recommendation.

The publisher may publish the unsolicited recommendation to a viewer orviewer set that is not currently subscribed, but which has granted thepublisher unsolicited publication privileges. Unsolicited publicationprivileges are an enhanced access right that a system may offer to itmembers. Basically, the system provides a means for exceptionalrecommendations to be posted to the users, even when they are notwatching TV or are not engaged in the system in the regular way. Forinstance, the system may post a text message of the given recommendationto a user that has presence status “online”. The publisher may post therecommendation to a subscribing viewer that is currently disabled or forwhom the publisher has privacy filters to normally block outboundrecommendation. Basically, the publisher temporarily overrides his ownoutbound privacy filters.

Target Filter

In some processing sequence alternatives, the outbound processing pathmay assume some or all of the target processing operations. Forinstance, a peer to peer system with thin subscriber component will tendto put the collaborative and target processing into the publishingcomponent. Target processing includes filtering by continuity, selectionpreference and subscription weighting. It also includes localization bycontent availability and viewing novelty.

Publication Settings

A viewer set may have default, minimum and maximum publicationconfiguration settings. This may be set up by the viewer setadministrator, and it is more likely to be used in a public communitywhere there are a large number of unfamiliar members. Publicationconfiguration settings may include any of the processing configurationsettings described in this section, plus the settings for the outboundgenerator node. For example, the Bay Area Historical Viewers Communitymay have a minimum publication requirement that requires all membersprovide their unedited, unannotated, anonymous viewing recommendations(while joined in).

Collaborative Processing

The collaborative processing node 1300 is generally the point wherepublisher recommendation inputs are aggregated into viewer setrecommendations. Then these viewer set recommendations are distributedto the recommendation subscriber targets. The collaborative processingmay be performed in the server, publishing clients, subscriber client,or a combination of these. (Some of the approaches described herein mayrequire a centralized collaborative processor, but some can bedistributed.)

Conventional collaborative filtering is the method of making automaticpredictions (filtering) about the interests of a user by collectingtaste information from many users (collaborating). This differs from themore simple approach of giving an average (non-specific) score for eachitem of interest, for example based on its number of votes. Relying on ascoring system which is averaged across all users ignores specificdemands of a user, and is particularly poor where there is largevariation in interest.

In potential systems there are scenarios with small variation ininterest and there are scenarios with large variation in interest. Thesystem may aggregate inputs in the collaborative processor to handleboth kinds of scenarios.

Pass Through

In buddy rosters and in close/familiar groups, there is typically asmall variation in interest amongst the members. The members are assumedto already be clustered as a collection with similar viewing interests.For these kinds of systems, the most appropriate collaboration may bethe straightforward scoring system averaged across all members.

In large viewer sets, the collaboration processor and/or thepresentation processor may have to average the input recommendations toprepare a more manageable recommendation notice to the subscribingviewer.

In small viewer sets, the collaboration may not even average the inputrecommendations. In these minimal systems, the collaborative processormay pass each recommendation, with publisher identification, through tothe subscribing viewers.

Community Collaboration

In unfamiliar groups and communities, there is typically a largevariation in interest amongst the members. In hierarchical communities,there may be larger variation. Full featured service providers (e.g.AOL-like service) may provide the opportunity for a viewer or viewer setto subscribe to anonymous cross-group recommendation viewer setcollections. In this scenario, unrelated groups or communities may pooltheir recommendations. These non-peer recommendations may be considered“supplementary”, “cross-group” or “world” recommendation sources. Theseunrelated cross-group viewer set combinations should have high variationin interest. For all these kinds of systems, conventional collaborativefiltering algorithms may be appropriate. Indices about the interests ofthe viewers and viewer sets may be extracted, and then predictions aboutthe interest of the member viewers and viewer sets can be generated. Themembers may be considered the individual viewers, or they may be thejoined/hierarchical viewer sets or they may be some combination ofthese. The interest indices can be generated directly from the contentpreference filter descriptions if they are managed by the system. Ifthese are not available, then the viewing UsageHistory for the membersmay be logged and examined to generate interest indices.

The collaborative filtering algorithms used in the system may be basedon a correlation coefficient which may be used to determine therelationship between different viewer's (or viewer sets') viewingpreferences. Alternative collaborative filtering algorithms may be used.

Statistical Aggregation

The methods described above (simple average, pass through, and communitycollaboration) may be configured with alternative statistical emphasisto produce different viewer set recommendations. Generally thecollaborative processor should tend to make recommendations that aremost similar in interest to the target subscribing viewers. Thecommunity collaborative algorithms use a combination of interest indices(preference filters) and publishing viewer endorsement (number of votes)to find the most similar selections. However, in many cases, the viewerset is assumed to be a collection of members with similar viewinginterests, and all published inputs are expected to be similar ininterest. In these cases, the average recommendation or the one with thehighest score may be considered main stream or typical for the viewerset.

There may even be a preference for atypical (outlier) programs that areviewed by very few members. This alternative statistical emphasis may bean inverted normal distribution of the published input endorsements. Forinstance, in addition to the collaborative statement “Most of yourfriends are watching program ABC”, the system may produce the statement“One of your friends has been watching program DEF, which no one elsehas watched yet”.

Collaborative Input

Collaborative processors which aggregate recommendations into rankedlistings may also take into account the context for each publishingviewer's input. The context influences the publishing viewer's selectionprocess, so it may also influence the collaborative process. Thepublisher context includes fixed elements such as the audio/video setupand the available content channels, plus dynamic elements such as otherviewers present and any factors affecting the publishing viewer'sattention to the content selection and consumption process. Theavailable content channels (or the available content program listing)may be incorporated in the collaborative process.

One possible approach to incorporate this type of context is to increasethe weighting of selections that are made from publishers with a broaderavailable content context. FIG. 14 depicts two publishing viewers withdifferent available content contexts. The selection recommendation fromviewer B is weighted slightly higher than viewer A's, because he has abroader set of choices.

A refinement to the approach above is to only adjust the selectionweighting when a particular recommendation input is not available to allpublishing viewers. In FIG. 15 A's and B's selection recommendations areweighted equally because they are available to all publishers. Howeverviewer C's selection recommendation is weighted slightly higher than A'sand B's because he made this choice after considering all of A's and B'scontent options plus the additional content choices he has. In theapproach depicted in FIG. 15 it may distribute a ranked listing ofrecommendations, because there may likely be a number of recommendationsthat are unavailable to some of the subscribing viewers.

The collaborative processor may accumulate the inbound recommendationsover time. This technique enables the processor to build a compositerecommendation listing that is refined in the temporal and collaborativedimensions. The composite listing benefits from the strengths of theinstantaneous recommendation processing, plus the strengths of theadditional processing dimension for accumulating over time.

Accumulation may be done before collaboration (outbound processor orpre-collaborative processor), after collaboration (post-collaborativeprocessor or inbound processor), at some intermediate processing pointwithin the collaborative process, or at some combination of these.

Accumulated recommendation listings are normally available to thesubscriber at any time, regardless of the activity of the publishingviewer set (e.g. if the subscriber is the only one watching TV).Typically, the subscriber might access or schedule the accumulatedlisting to be presented at a regular time. For instance, the subscribermay use the accumulated listing every Friday night, to see what contentrecommendations have become noteworthy over the preceding week.

The subscriber's Friends Recommend listing and other list-type displaysmay contain an addendum or side column of the accumulatedrecommendations. These list displays may also merge the accumulatedrecommendations with some appropriate designation for accumulatedstatus. The accumulated recommendation listing is a dynamic (living)collection that is actively updated by the system according to someappropriate update mechanism. The subscriber list-type displays (e.g.Friends Recommend) may be updated instantly, as the accumulatedrecommendation listing changes.

Inbound Processing

The subscribing component may further process the inbound recommendationdata. This inbound processing includes filtering and localizing therecommendation. There are a number of sub processing steps that may beused for filtering and localizing. These operations may be co-located orthey may be distributed over the subscribing inbound processor, thecollaborative post-processor, the collaborative pre-processor, or eventhe publishing outbound processor. Generally, these operations areassociated specifically with the subscribing viewer, so they aredescribed in this section. The collaborative post-processor stage isdescribed implicitly by these operations, and that stage is notdescribed further in this document.

Out-of-Preference Threshold

The inbound recommendation data is filtered to remove unwanted items.The filters are based on configuration preferences defined for thetarget (subscribing) viewer. Inbound recommendation data may be filteredto insure it is within a desired criteria. The target subscribing viewermay use his regular content preference filters or a specialized inboundcontent preference filter 1320. As with most preference filterdescriptions, an appropriate time, location and device condition set maybe used to qualify the use of the preference filter. (In MPEG-7 andTV-Anytime, this is the preference condition description scheme.) FIG.16 depicts a set of content filter descriptions for a system that usesdedicated out-of-preference and in-preference descriptions.

If the subscribing viewer's content management system has a technique ofdetermining disinterest/non-preference for a program (e.g.“thumbs-down”), the preference out-of-filter description (regular orspecialized) may be dynamically updated to reflect these non-preferreditems as they are graded by the subscriber.

The out-of-preference filtering process may use the degree of preferencefor the recommendation, the degree of preference match (or mismatch) tothe criteria, and the out-of-preference threshold. This will reduceundesired programs but still allow some marginal cases to be passed. Forexample, it might permit a strong recommendation for opera, even if yourpreferences show a slight rejection for that category). Programs thatare not removed may have their recommendation preference value modifiedor augmented with the degree of preference match.

In Preference Reverse Threshold

Inbound recommendation data is filtered to remove readily apparentprogram selections. The target subscribing viewer may use his regularcontent preference filters or a specialized inbound content preferencefilter (separate from the out-of-preference specialized filter above).The filtering process may use the degree of preference for therecommendation, the degree of preference match to the criteria, and thein-preference reverse threshold. Programs that are not removed may havetheir recommendation preference value modified or augmented with thedegree of preference match.

Program Continuity

Inbound recommendation data is filtered to remove short durationrecommendations at process 1310. This is especially important to filterout channel surfing. A minimum duration threshold (e.g., temporal) isused to remove or hold back recommendations. If the implementation useschange of program messages to publish the raw recommendation data, thenthis filter may hold back the recommendation for the defined duration,and pass it through once the period has elapsed. If a new program changeis posted before the period elapses, then the old message is deleted andthe timer is reset. There may be a publishing program continuitythreshold as well as a subscribing program continuity threshold. Thepublishing filter will use a minimal threshold to help reduce change ofprogram message traffic. The subscribing filter will refine and expandthe continuity threshold to meet the subscribing viewer's needs.

Various continuity threshold values may be applied at the subscribingviewer to suit the viewing mode of the subscriber. For instance, a longduration threshold may be used when the viewer is immersed in a program,but a short duration threshold may be used when he is looking at a“Friends Are Watching . . . ” type of user interface.

Publisher Weighting

Inbound recommendation data is modified to reflect the subscriber'sweighting of the specific publisher. For instance, “John'srecommendations are more pertinent than Jerry's”. The subscriber inboundprocessor maintains a database of one or more publishers with associatedpublisher weightings at process 1330. This value is applied (e.g.multiplied) to inbound recommendation preference values from the givenpublishers. A default weighting (e.g. 1.0) is applied to inboundrecommendations from publishers that are not in the database. Thepublisher weighting database may also be a simple inbound permissionfilter. It can filter out specific publishing viewer recommendations.The underlying presence infrastructure may support blocking (e.g. notconnecting to) individual publishing viewers. However, in an organizedviewer set (group or community) the subscriber does not connect toindividual publishers, so he preferably does not set permissions forindividual inbound recommendations. The publisher weightings may beapplied anywhere in the processing stream where this database isavailable. The publisher weightings may be applied for a viewer setrather than for a specific subscriber. In this case, the weightingsshould be applied before the collaborative processing.

Localized Recommendation

Localization 1340 inserts the inbound recommendation into the context ofthe subscribing viewer. The inbound recommendations may be processed toassemble them with associated electronic programming guide, localscheduling, and/or UsageHistory, to build the localized/personalizedrecommendation. Typically, the localization is done in the subscribingcomponent, where the context is available. However, if the server orother module is aware of the subscriber's context, then it may performthis processing.

Content Availability

The inbound recommendations are filtered to remove items that areunavailable to the subscribing viewer. Alternatively, the items may bedenoted as unviewable for the subscriber user interface (e.g. in the“Friends Are Watching . . . ” user interface). The listing of locallyavailable content is maintained by the content management system. Thismay include electronic program guide (or other scheduling service)personal video recorder, and local media recording servers, Internetcontent provider guides, and other stored content guides. Programs thatmay be available on a pay-per-view or on-demand basis may be designatedas conditionally available. Programs that are not presently available,but will be available at a later time may be designated with upcomingavailability. Programs that are not immediately available may bepostponed, or may have an attribute of reduced urgency applied to them.Reduced urgency items would generally not be displayed to the viewerunless they are actively navigating the recommendations user interface.

Content Novelty

The inbound recommendations are filtered to remove items that arecurrently, recently or previously viewed. Alternatively, the items maybe denoted as currently/previously viewed for the subscriber userinterface (e.g. in the “Friends Are Watching . . . ” user interface.)The UsageHistory log contains details about viewed programs. If theinbound recommendation is one of a program series or other relatedprogram collection, then the content novelty processor may check forboth the specific recommended episode/instance of the program as well asthe containing series/collection. Program episodes that have not beenpreviously viewed but whose containing series/collection has beenpreviously viewed may be denoted as series previously viewed. If theinbound recommendation input has a timestamp or other program offset toindicate which point within the program is being recommended, then thenovelty of the program may be determined at the sub-program level. Inthis case, the content novelty process may use the browsing details fromthe UsageHistory log to determine if the viewer has viewed therecommended portion. So, even if the viewer has watched a differentportion of the program, making that portion non-novel, there may stillbe a novel portion of the program that should be recommended to thesubscriber.

Presentation

The presentation 1350 and interaction processes 1360 make up theextended user interface. This includes the preparation, display, andinteraction of the recommendations.

The inbound subscribed recommendation should support the source dataelements and it should also support the following elements and concepts.

-   -   Recommendation available—status indication for the        recommendation. Typically this invokes the initial notification        to the subscribing viewer.    -   Endorsement—description of how the item is recommended. This may        include a list or count of recommending entities, a description        of the accumulated or instantaneous nature of the recommendation        and/or the degree of preference.

Data Elements

The inbound degree of preference may be a composite or individuallisting of the degrees of preference for all endorsing publishers.Endorsements may be negative, as depicted by a negative degree ofpreference. The negative recommendation, especially one for a programthat the publisher has watched for a lengthy amount of time, is animportant input to the subscribing viewer set. If the publisherconsiders the program selection to have been a poor choice or a mistake,then it is also likely that the peers of the publisher would make thesame selection mistake and would want to be alerted to the potentialpoor choice. Special handling may be provided for negative endorsements.

A negative endorsement for a program that several members of a viewerset are watching may result in an immediate alert to all those watchingit. This may help the viewers to more quickly decide to exit anundesirable program.

Special icons may be used to designate negative preference whereverrecommendations are displayed (e.g. the “Friends Recommend” display).

-   -   Endorser status—description of how the item is currently being        consumed by the recommending entities. This may include        indication that viewers are currently watching or have stopped        watching.    -   Endorser supplementary data available—description of data and        services that may be available from the recommending entities.        This may include simultaneous conversation communication        sessions for this selection, or active browser remote        controller, etc.    -   Program supplementary data available—description of data and        services that may be available for the specified program        selection. This may include summarization description,        alternative content delivery methods, commercial recommendations        and links.

Display

Recommendations are displayed to the user in a variety of ways.Conceptually, these may be different types of on-screen displays ordialog boxes.

Notification

Notification that recommendations are available should be the earliestand lightest display. Notification display should be tailored to theviewing mode of the user. A different strategy and condition thresholdshould be used when the viewer is immersed in a program versus when heis actively navigating some control functions. Also, notificationdisplay should be configurable to adjust the type, urgency, threshold,etc.

Notification may be performed immediately by a momentary icon displayedon the screen, or an audible beep, or by other unobtrusive method.Notifications may be accumulated for less frequent interruptions. Forinstance, recommendations may be collected for a period of time or for aminimum count threshold, and then a notification may be rendered to theviewer. Notifications may be accumulated and delayed until triggered byviewer interaction. For instance, when the viewer accesses any controlfunctionality, a notification of available recommendations may berendered.

The complete recommendation detail listing may be displayed when theviewer purposefully chooses to examine all the details. This may containall the published data elements or the inbound subscribed data elements.This display view is typically not the first view offered for arecommendation, rather, it is offered as a secondary option available inmost of the other display views listed below.

Attached to Guide

Recommendation data is an extension to any program guide or otherscheduling guide. Each program listing that has associatedrecommendations available should display an available recommendationsindicator with the listing. For instance, an icon or special charactercan be displayed on the line for the program. Other data elements mayoptionally be displayed directly in the program listing, including thedegree of endorsement, the availability of supplementary data, etc.Finer details for the recommendation should be accessible by selectingan option on the program listing, which brings up the recommendationdetail listing display.

Friends

This display view lists all the recommendations that are targeted andapplicable to the subscribing viewer. This should be a complete listingof all the valid recommendations in the subscriber's system. It shouldonly list these valid recommendations, and exclude other availableprograms (that are not recommended) and other invalid recommendations(that are not available).

The Friends Recommend listing may be organized by the publishers(friends) or it may be listed by the recommendations. (The user may havethe option to alternate between these two ordering schemes.) The detailthat is offered in this display view may be a combination of programdescription and endorsement details. Finer details about the program maybe available through extended program detail listings (i.e. displayoptions offered natively by the device) and finer details about therecommendation may be available through the recommendation detaillisting display.

This display view lists the current viewing selections of allparticipating publishing viewers. This listing should be different thanthe Friends Recommend listing because it includes invalidrecommendations (that are not available), and it may include surfingprogram selections. Also, it should exclude old recommendations andprogram selections which are not currently being watched. This should bean instantaneous picture of what everyone in the viewer set is watching.Because of its instantaneous nature, target filtering and localizationprocessing should be bypassed for this view.

Program selection listings that are invalid should be denoted with anappropriate emphasis in the display. For instance, a publisher'sselection that is invalid due to the continuity filter may have an iconof a surfer on the same line. This concept of a surfing indication isanalogous to the typing indicator in AOL Instant Messaging. The FriendsAre Watching listing may be organized by the publishers (friends) or itmay be listed by the current program selections. (The user may have theoption to alternate between these two ordering schemes.)

Quick Pick

The sorted recommendation list or a subset of the most preferred entriesof this list may be displayed in a quick pick listing. This may be anexclusive listing that is tied to a dedicated control button, or thelisting may be combined with the “Favorite Channels” listing that isalready maintained on the device. Preferably, the FAVORITE controlbutton or similar would display the predefined list of favoritechannels, and appended or inserted in the list are the several mostpreferred programs from the recommendation list. The combined listing isa circular queue which the viewer may scroll up or down, and wraparound. The listing should be limited to a number of channels/programsthat can be displayed on one screen.

In addition, the recommendations may be displayed in suitable on-screennotification. The recommendations may be attached to program guides,where a portion of the program guide indicates the recommendation. Theportion of the program guide indicating the video may include anindication that the particular video is recommended, and also the levelof recommendation (e.g., from low to high). Other presentationtechniques may likewise be used, such as for example, displayed inrecommendation based listings, displayed in persistent listings,displayed in full-detail listing, and attached to browser controldisplays.

The inserted recommendation may be displayed with program descriptioninformation, or it may be displayed as just the content channelindex/mnemonic. If it is displayed similar to the favorite channelsformat, then it should be augmented with some emphasis (e.g. icon, textnote) to indicate the channel is carrying a recommended programselection.

Persistence

A dedicated region of the screen may show persistent listing of theFriends Recommend or Friends Are Watching display views. The data in thedisplay would be instantly updated as the recommendation list changes.The listing may be organized by the publishers (friends) or by therecommendations. If it is organized by the publishers, it may be modeledafter presence status displays by showing all members with their currentstatus. The display region may overlay the current program or it mayoccupy a side/top/bottom bar region. The bar region may overwrite thecurrent program or it may compress the video to fit in the reduceddisplay area.

Browsing

Onscreen browsing controls may be augmented with recommendationinformation. In particular, the current viewing timepoints/offsets ofthe endorsing viewers may be superimposed onto the browsing controldisplay. The current viewing timepoints/offsets of endorsing viewers area part of the endorser position data in the recommendation listing. Thisdata should be updated instantaneously as the publishing viewers browsewithin this program.

Others

Outbound recommendations may be displayed for editing, routing,confirmation, and information purposes. Generally, there may be a seriesof recommendation messages associated with any change in the contentconsumption. This will include program changes, program selectionstabilization, browsing operations, and other navigation functions. Apersistent onscreen display can depict the status and other details ofthe outbound recommendation stream. Major changes in the recommendationmessages (e.g. change of program, stabilized program selection) maytrigger an onscreen dialog or an optionally selectable menu forexamining and editing the data. Recommendations may also be initiatedand edited directly from the control menus, which allows the viewer tomanually generate recommendations for the system.

Action

Within the context of the various recommendation display views, the usermay have several types of interactions available.

-   -   Further info—The viewer can probe for further details or a        different view of the recommendation data. For instance, the        viewer may receive an onscreen notification that recommendations        are available, which may cause him to navigate through a control        menu to retrieve the Friends Recommend listing and then dig        deeper to a Recommendation Detail listing.    -   Instant select—Within any of the recommendation listings, the        viewer may select a listing and then choose the instant select        control action (e.g. the play button) to switch to that content        channel or recorded media source.    -   Defer action—The viewer may decide to postpone the        recommendation for future viewing. A control option may be        displayed to indicate that the content is recorded, can be        recorded, or is available in a future broadcast run. This allows        the viewer to quickly accept and react to a recommendation,        without having to completely stop what he is doing.    -   Enable service—The viewer can act upon supplementary control        options that may be available. If a summarization description is        available or a browsing remote control session is active for the        given program, then the viewer can select the appropriate option        to pass control to the supplementary control item.

User Interface

The user interface can be implemented in many ways that are appropriateand differentiating for the application. This section describes someconcepts that are relevant to the user interface implementation.

-   -   Each presentation item may be considered solicited (e.g. menu or        control button initiated) or unsolicited (e.g. popup alert).        Some of the items may be configured with a degree of this        characteristic. For instance, notifications can be completely        unsolicited and immediate by using popup alerts, which provides        the best degree of alert. Alternatively, notifications can be        completely solicited by embedding in menus, which provides the        best degree of unobtrusiveness. Preferably, the configuration        allows an appropriate alert/unobtrusiveness level setting that        will make notifications be a combination of the two extremes,        which provides an intermediate degree of alert and        unobtrusiveness.    -   Presentation items may have a degree of occlusion. Onscreen        displays that overlay or compress the video will occlude or        reduce the visibility of the program.    -   Presentation items which list multiple pieces of information may        have a degree of information detail. Generally, it is better to        minimize the information displayed to simplify use of the system        (and reduce the degree of occlusion).    -   Audible presentation may include tones, beeps, voice synthesis,        etc.    -   Graphical presentation may include icons, text, tables, roster        lists, scrolling text, video, images, etc.

Remote Control

The presence technology infrastructure provides the basis for sharinginformational messages that support the system recommendations. Thisinfrastructure can also support control type messages that can be usedto coordinate operations on remote viewers' devices. One leader ormoderator may control several other subscribing viewers' devices.Alternatively, several publishing viewers or the whole viewing set maycollaboratively control the subscribing viewers' devices.

The subscribing viewer can elect to let his device's program selectionbe controlled by one individual publisher or the publishing viewer set'scollaborative recommendation. Effectively, the subscriber links theirtuner selection control to the network. This remote control providesimmediate actions on the subscribing device, for currently availablebroadcast or mutually recorded content.

The subscribing viewer can configure their device to record programselections from the viewer set. The subscriber may want the device toautomatically record program recommendations from the viewer set. Theviewer set is building the suggestion list on this individualsubscriber's local machine. If the viewer is out of town or not watchingTV, the viewer set will prepare his content so that he may get up todate when he returns.

The capture remote control is that the device may include an entitypresence that is separate and unique from the viewer's presence. Thisdistinction is important when the viewer and the device are not actuallyoperating on the same content. For instance, if the viewer is notwatching TV, or if the viewer is watching content while the device isscheduling/recording different content. Presence for the device shouldhave a similar meaning as for the human viewer. Status should indicateif the device is online or offline. Publishing input should indicatewhat the device is recording or changes in its recording schedule.

The remote control may simply schedule to record inbound programrecommendations that meet a predefined criteria. For instance, programswith a preference match value exceeding a given threshold would bescheduled. The remote control may have more sophisticated conditions foraccepting the inbound recommendations such as a minimum number orpublishing recommenders, or a specialized content preference filterdescription. The remote control may have configuration settings thathandle recording space restrictions. For instance, recording is onlyenabled when there is available space or when the first-to-delete(lowest priority) program is non-sports and the recommendation is for asports program.

The remote control may be performed with a separate collaborativeprocess that is performed by the publishing device presence entities.The involved devices may interact directly, or a central collaborativeprocessor may coordinate the decision. Preferably, the collaborationprocessor would calculate some index of mutual preference. The mutualpreference index would tend to favor maximizing the occurrences ofdevices recording the same content, but allows for individuals orclusters of individuals to record different programs according to theassociated viewer's content preference filters. For example, severalbaseball fans may link their devices in capture remote control. Duringthe middle of the night when the viewers are not watching or schedulingcontent, multiple foreign baseball games become available from theircontent providers. The devices will cast their recordingrecommendations. If there are conflicts which prevent recording some ofthe recommended games, then the collaborative process will select thenon-conflicting program sequence with maximum mutual preference index.By prior agreement, all the devices schedule this program sequence forrecording. The following day, the baseball fans will have a common setof baseball games which they may consume for mutual discussion, orpreferably, co-view with supplementary conversational communication andshared device browsing remote control.

The subscribing viewer can configure their browsing (transport) to beunder the remote control of the system. Skipping ahead, repeating clips,playing at slow or fast rate and other browsing operations may becontrolled remotely. Remote program selection may also be part of remotebrowsing control.

The controlling entity is generally a designated publishing viewer.Effectively, the controller's browsing control actions are broadcast tothe subscribing viewers. Alternatively, the controlling entity could bea previously generated program summarization description which thesubscribing viewers agree to consume simultaneously. (If the viewersconsume the summary in an uncoordinated non-simultaneous manner, thenthey are not participating in browsing remote control—rather they arejust individually consuming the summarization description.) A livebrowsing remote control session will have a single designatedcontrolling publisher at any point in time. This controller designationmay shift from viewer to viewer, similar to passing the TV remotecontrol around while watching TV as a group.

The preferred content requirements for browsing remote control aresimilar to those for program summary recommendations. The content shouldbe non-linearly navigable with consistent timestamp/offset information,and it should be available to both the publishing and the subscribingviewers. Additionally, the content should be available at the same time(and on the same run).

The hardware (and controlling software) should support the transportcontrol messages. The control should be nearly instantaneous andcoordinated with the timestamp or offsets as described previously.

To allow for the non-zero latency of the control messages, there may bea small time shift inserted in the playback of the subscribing viewers.The publisher is watching the program slightly ahead of the subscribers.The shift is made long enough that when the publisher performs abrowsing action at a particular point in the program, the controlmessage may be generated, broadcast, received and interpreted prior tothe subscribers' devices reaching the same point in the program. Simpleor sophisticated techniques can be used to warp the playback rate sothat the viewers are watching nearly the same program. Minimizing thetime shift (and warping) may be important to keep the playback browsingsynchronized with the supplementary conversational communication.

Communication Channels

The system may be extended to support supplementary conversationalcommunication channels. While viewers are sharing content selection andcontent consumption recommendations, they may also interact in a liveconversation session. The conversation sessions may be triggered bysystem activity, or the sessions may precede the activity.

The communication channels may match up with the paths ofrecommendation, or they may exist on a subset or superset of the paths.FIG. 17 depicts the components of a system extended with conversationalcommunication channels. The presence network and the communicationnetwork may share network and infrastructure resources, or they may beseparate (as shown).

Instant messaging is a primary form of presence technology that isdirectly usable as a supplementary communication channel. It has a wellestablished set of protocols and implemented applications with richsupport for conversational communication sessions. Presence technologymay also be used to support new conversation channels that are outsideof instant messaging. The Jabber protocol and other protocols whichsupport presence can be used to deliver new structured message formatsand new services that can implement the supplementary communicationchannel extensions from the system. This alternative presence basedformat may be the preferred way to support custom viewer representationsand custom conversational interaction.

Conversational communication sessions can be implemented using otherinternet based protocols without presence technology. These may includevoice over IP, videoconferencing over IP, web data conferencing,place-based collaboration environments (e.g. Microsoft NetMeeting).Conversational communication sessions can be implemented over othercarriers such as the content provider channel. Interactive TV (ITV) andWeb TV are established protocols that are operational in some market.These infrastructure systems provide 2 way communication within the usercommunity.

When a conversation session is initiated on a supplementarycommunication channel, the users may be offered various selfrepresentation models. This model is a depiction of the viewers, used touniquely identify them in the conversation. Audio oriented communicationchannels (e.g. voice over IP, voice conferencing) may rely on theidentifiable voice of each member. Alternatively, a set ofdifferentiating tones or beeps could be used to identify eachcontributor. Text oriented communication channels (e.g. text IM, chatrooms) will probably use the nickname or handle of each member. Videooriented communication channels (e.g. videoconferencing over IP) may usea variety of self representation models, including live video, iconicgraphics, picture images, maneuverable images, or some combination ofthese.

Live video may be supported by standard web cam or PC cameras focused onthe participating viewers. Static icons may be selected from apredefined list or users may generate their own. Similarly they mayinstall a static image. Maneuverable images may be supported by MPEG-4standardized Facial Animation Parameters. Also, static and maneuverableavatars may support a range of expressions and gestures in theconversation. These two maneuverable visual representations may be anattractive feature of the communication channel—simultaneously allowingprivacy and flexibility of expression. The self representation modelsmay be mixed and users may dynamically change their model to meet thechanging needs of the conversation.

The interaction supported by the conversational communication sessionshould support regular discussion and commentary. In enhancedimplementations, the conversation may be integrated into other aspectsof the system.

-   -   Systems which support browsing remote control may use the        parallel conversation channel to augment the browsing and to        implicitly manage the hand-off of browsing control from one        viewer to another. Through verbal or visual indications, a        request for browsing control can be submitted and then accepted        or rejected. (e.g. John says, “let me drive for a bit here        Larry”.)    -   Systems which support place-based collaboration environments        (e.g. Microsoft NetMeeting) will use the parallel conversation        channel to augment the collaboration environment. (In fact, the        parallel conversation channel is an integral part of these        systems.) Through on screen pointers, a viewer can emphasize or        sketch notes on the concurrently viewed program.    -   Systems which support live interactive collaborative selection        will use the parallel conversation channel to support and        coordinate the selection process. In this scheme, a selection        menu (e.g. EPG guide or keyword search) is synchronously        presented on all the participating members' devices. Through the        conversation channel, the users may describe what they are doing        as any of them controls or modifies the selection interface.

Supplementary Recommendations

The pure recommendations generated by the device may be augmented withother related recommendations to enhance the system functionality.

The recommendations generated by linking to anonymous cross-group viewersets are a type of related recommendation that may be provided as asupplement to the regular in-group sources. These cross-group items andall the related recommendations described in this section may bepresented to the viewer in a side panel or “additional” navigationbutton on any of the user interfaces that show the recommendationlistings. For instance, the “Friends Recommend” user interface may havethe pure listing on the left side of the screen, and a separate listingon the right containing the related recommendations.

The recommendations that are passed through the localization processorare known to be available to the viewer. The related recommendationslist may be implemented so that program selections that are notimmediately available, but may be available in VOD or other pay-per-usemethods, are presented in the related recommendations list. This list ofVOD-available selections would preferably be generated by the regularpublishing viewer set. Additionally, it would contain cross-grouprecommendations and the wholly-commercial collaborative results ofrecommendation engines.

The list of presently unavailable programs that is presented in therelated recommendations user interface may be a service to help theviewer consume the content they desire, or it may be a means to pull theviewer into new revenue-paying content that they might not have viewedotherwise. An extension of the straight content delivery options is theprovision for alternate content versions. For instance, the contentdelivery options under the related recommendations list may containdirector's cut versions or high definition versions of the same program.

Related recommendations may include links and summaries that enhance thecontent consumption experience. These may be offered as options to theregular content playback. The interactive capability that is integratedin the various recommendation listings may offer selectable options thatenable one or more of the following.

-   -   Program summarization description—e.g. a professionally        generated summary for a sports event    -   Program commentary, summary notes, background material    -   Related links, broadcaster web pages, program web pages—e.g.        provided through a TV or compliant format.

IP Television

Other configurations may include a video display device, such as an IPbased television, that permits the bidirectional data communications. Inthis configuration, the local storage of video content would be replacedor otherwise supplemented by a network storage device. In some cases,the display device may be in the form of a game console.

1. A system comprising: (a) a first audiovisual device capable ofreceiving audiovisual content; (b) a second and third audiovisual devicecapable of providing data descriptive of said audiovisual content; (c)said first audiovisual device obtaining recommendations for audiovisualcontent by said second and third audiovisual devices, which eachcategorize said recommendations under a hierarchial relevance scheme;(d) said first, second, and third audiovisual devices joined in a firstgroup; (e) a fourth and fifth audiovisual device capable of providingdata descriptive of said audiovisual content; (f) said first audiovisualdevice obtaining reccomendations for audiovisual content by said fourthand fifth audiovisual devices, which each categorize saidrecommendations under said hierarchial relevance scheme; (g) said first,fourth and fifth audiovisual devices joined in a second group; (h)wherein said first audiovisual device filters recommendations using saidrelevance scheme, and wherein recommendations provided by said firstgroup are filtered by said first audiovisual device in a differentmanner than are recommendations provided by said second group.
 2. Thesystem of claim 1 wherein said filtering is based upon different privacycharacteristics.
 3. The system of claim 1 wherein said filtering isbased upon feedback from said first audiovisual device.
 4. The system ofclaim 1 wherein said first audiovisual device is capable of providing arecommendation for audiovisual content to a third group to which saidfirst audiovisual device is free from being joined.